Calipers performance25 by mo-marqh · Pull Request #176 · MetOffice/lfric_apps

mo-marqh · 2026-01-22T15:14:47Z

PR Summary

Sci/Tech Reviewer: @tommbendall
Code Reviewer: @oakleybrunt

Comprehensive timing caliper location review.

This is a rework of work done for performance analyses in late 2025, migrating everything caliper related from:
https://code.metoffice.gov.uk/trac/lfric_apps/browser/main/branches/pkg/Share/r15393_kpi_benchmark
essentially from this commit:
https://code.metoffice.gov.uk/trac/lfric_apps/changeset?reponame=&new=15498%40main%2Fbranches%2…

but updated from source migration and adoption of timing wrapper from #80

is related to Verniered Calipers performance 25 core lfric_core#233 (but these are not linked, each is stand-alone)

Code Quality Checklist

I have performed a self-review of my own code
My code follows the project's style guidelines
Comments have been included that aid understanding and enhance the readability of the code
My changes generate no new warnings
All automated checks in the CI pipeline have completed successfully

Testing

I have tested this change locally, using the LFRic Core rose-stem suite
If required (e.g. API changes) I have also run the LFRic Apps test suite using this branch
If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - calipersPerformance25/run6

Suite Information

Item	Value
Suite Name	calipersPerformance25/run6
Suite User	mark.hedley
Workflow Start	2026-01-22T19:51:59
Groups Run	all

Dependency	Reference	Main Like
casim	MetOffice/[email protected]	True
jules	MetOffice/[email protected]	True
lfric_apps	mo-marqh/lfric_apps@calipersPerformance25	False
lfric_core	MetOffice/lfric_core@aa32824	True
moci	MetOffice/[email protected]	True
SimSys_Scripts	MetOffice/[email protected]	True
socrates	MetOffice/[email protected]	True
socrates-spectral	MetOffice/[email protected]	True
ukca	MetOffice/[email protected]	True

Task Information

✅ succeeded tasks - 1456

Security Considerations

I have reviewed my changes for potential security issues
Sensitive data is properly handled (if applicable)
Authentication and authorisation are properly implemented (if applicable)

Performance Impact

Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

I understand this area of code and the changes being added
The proposed changes correspond to the pull request description
Documentation is sufficient (do documentation papers need updating)
Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

All dependencies have been resolved
Related Issues have been properly linked and addressed
CLA compliance has been confirmed
Code quality standards have been met
Tests are adequate and have passed
Documentation is complete and accurate
Security considerations have been addressed
Performance impact is acceptable

mo-marqh · 2026-01-26T09:09:20Z

timer.txt
example timer.txt from /cylc-run/calipersPerformance25/run6/trac.log

✅: timer'txt'

Routine	min time(s)	mean time(s)	max time(s)	No. calls	%time	time per call(s)
lfric_atm	343.66	343.86	346.68	1	100.0000	343.8605
setup	31.87	31.90	31.93	1	9.2762	31.8972
gungho_driver.initialise	31.87	31.90	31.93	1	9.2762	31.8971
runtime_constants.geometric	1.49	1.53	1.72	23	0.4442	0.0664
mappings.set_wind	1.55	3.98	6.74	145	1.1564	0.0274
runtime_constants.fem	6.46	6.89	7.22	29	2.0049	0.2377
runtime_constants.mapping	0.30	0.30	0.33	1	0.0884	0.3041
runtime_constants.dycore	27.58	27.90	28.32	13	8.1135	2.1461
mass_matrix_solver_alg	5.49	5.58	5.64	181	1.6233	0.0308
mappings.map_physics_fields	4.24	4.31	4.36	37	1.2536	0.1165
gungho_diagnostics_driver	38.68	39.08	39.24	37	11.3658	1.0563
gungho_driver.first_timestep	44.46	44.46	44.47	1	12.9307	44.4637
gungho_timestep	271.53	271.71	272.15	36	79.0170	7.5475
semi_implicit_timestep	271.53	271.70	272.14	36	79.0160	7.5474
dynamics.compute_si_operators	20.76	21.37	21.46	18	6.2155	1.1874
runtime_constants.solver	0.01	0.02	0.02	17	0.0044	0.0009
slow_physics	107.74	107.76	107.79	36	31.3395	2.9935
cloud.fsd_condensate	0.09	0.10	0.11	36	0.0294	0.0028
locate_tropopause	0.01	0.01	0.01	36	0.0020	0.0002
diags.locate_tropopause	0.00	0.00	0.00	36	0.0001	0.0000
aerosol.glomap_clim	2.90	3.00	3.31	36	0.8712	0.0832
diags.glomap_clim	0.00	0.00	0.00	36	0.0011	0.0001
aerosol.stratosphere	0.00	0.00	0.01	36	0.0012	0.0001
microphysics.wb	4.65	7.96	11.22	36	2.3138	0.2210
diags.mphys	5.76	7.41	9.07	72	2.1552	0.1029
radiation.illuminate	0.02	0.02	0.02	36	0.0066	0.0006
diags.illuminate	0.00	0.00	0.00	36	0.0004	0.0000
aerosol.radaer	22.72	26.68	30.27	18	7.7596	1.4824
diags.radaer	0.00	0.00	0.00	54	0.0008	0.0001
jules.radiation	0.25	0.51	0.69	36	0.1482	0.0142
diags.radiation	23.34	23.36	23.38	72	6.7923	0.3244
radiation.sw	3.25	4.67	6.87	36	1.3584	0.1297
radiation.lw	10.09	11.67	12.38	36	3.3928	0.3241
cloud.pc2_radiation	0.22	0.23	0.26	36	0.0676	0.0065
diags.pc2_radiation	0.00	0.00	0.00	36	0.0004	0.0000
spectral_gwd	0.54	0.57	0.69	36	0.1658	0.0158
diags.spectral_gwd	0.00	0.00	0.00	72	0.0009	0.0000
orographic_drag	0.05	0.27	0.55	36	0.0775	0.0074
diags.orog_gwd	0.00	0.00	0.01	72	0.0011	0.0001
methane_oxidation	0.00	0.00	0.00	36	0.0011	0.0001
diags.methox	0.00	0.00	0.00	36	0.0001	0.0000
jules.explicit	7.20	13.70	18.00	36	3.9846	0.3806
diags.jules_exp	3.14	3.47	3.79	72	1.0079	0.0481
bl.explicit	1.72	2.62	3.66	72	0.7617	0.0364
runtime_constants.physics	0.03	0.04	0.05	1	0.0117	0.0404
diags.bl_exp	0.00	0.01	0.01	72	0.0017	0.0001
cloud.bm_tau	0.21	0.23	0.25	36	0.0659	0.0063
cloud.pc2_checks	0.22	0.23	0.29	36	0.0666	0.0064
diags.pc2_checks	0.00	0.00	0.05	36	0.0011	0.0001
dynamics.rhs_alg	15.14	15.19	15.26	180	4.4177	0.0844
dynamics.transport_predictors	2.66	2.74	2.78	36	0.7971	0.0761
dynamics.transport	33.02	33.10	33.14	72	9.6256	0.4597
transport.controller_init	2.09	2.16	2.19	72	0.6274	0.0300
runtime_constants.transport	0.01	0.01	0.01	6	0.0031	0.0018
transport.flux_precomp_init	0.10	0.11	0.14	576	0.0306	0.0002
transport.flux_precomp	1.86	1.96	2.08	1692	0.5703	0.0012
transport.hori_dep_pts	1.35	1.46	1.56	216	0.4260	0.0068
control_density_transport	2.23	2.26	2.28	72	0.6568	0.0314
transport.ffsl_vertical	7.19	7.29	7.48	2232	2.1201	0.0033
transport.ffsl_horizontal	12.91	13.14	13.25	684	3.8221	0.0192
transport.wind_to_comps	0.57	0.60	0.64	72	0.1736	0.0083
transport.sl_horizontal	3.59	3.76	3.83	432	1.0947	0.0087
transport.wind_from_comps	3.28	3.31	3.33	72	0.9616	0.0459
control_moisture_transport	7.43	7.47	7.50	72	2.1718	0.1037
control_advective_tracer_transpo	5.62	5.66	5.77	72	1.6462	0.0786
control_conservative_tracer_tran	0.00	0.00	0.00	72	0.0001	0.0000
control_potential_temperature_tr	1.97	2.08	2.17	72	0.6037	0.0288
transport.wind_precomp_final	0.00	0.00	0.00	432	0.0002	0.0000
dynamics.phys_predictors	3.40	3.43	3.47	72	0.9979	0.0477
fast_physics	46.22	46.29	46.32	72	13.4611	0.6429
convection.gr	7.57	11.16	15.21	72	3.2460	0.1550
diags.conv	6.00	7.05	8.02	108	2.0509	0.0653
cloud.pc2_conv	0.22	0.26	0.41	72	0.0755	0.0036
diags.pc2_conv	0.00	0.00	0.00	72	0.0007	0.0000
bl.implicit	15.41	15.56	15.73	144	4.5247	0.1080
diags.bl_imp	3.86	3.94	3.97	108	1.1446	0.0364
jules.implicit	0.10	0.18	0.27	144	0.0515	0.0012
dynamics.solver	19.75	19.80	19.84	144	5.7577	0.1375
solver.mixed_solver	18.87	18.93	18.98	144	5.5053	0.1315
mixed_solver.schur_precon	8.82	8.95	9.01	509	2.6015	0.0176
schur_precon.rhs	0.66	0.69	0.73	509	0.2012	0.0014
schur_precon.pressure_solver	6.72	6.84	6.94	509	1.9902	0.0134
pressure_solver.helmholtz_lhs	4.09	4.25	4.35	7635	1.2348	0.0006
schur_precon.back_sub	1.20	1.41	1.50	509	0.4090	0.0028
mixed_solver.operator	3.49	3.67	3.96	509	1.0676	0.0072
control_last_advective_tracer_tr	0.97	1.05	1.09	36	0.3051	0.0291
control_last_conservative_tracer	3.97	4.05	4.14	36	1.1768	0.1124
diags.jules_imp	11.66	11.90	11.97	72	3.4601	0.1653
jules.extra	0.03	0.10	0.22	36	0.0301	0.0029
diags.jules_snow	3.44	3.51	3.55	72	1.0211	0.0488
diags.jules_soil	0.00	0.00	0.01	72	0.0011	0.0001
diags.jules_seaice	0.00	0.00	0.00	36	0.0006	0.0001
cloud	1.10	1.19	1.42	36	0.3449	0.0329
diags.cloud	9.94	10.12	10.16	72	2.9440	0.1406
cloud.pc2_pressure	0.24	0.26	0.27	36	0.0756	0.0072
diags.pc2_pressure	0.00	0.00	0.00	36	0.0004	0.0000
cloud.pc2_initiation	0.81	0.88	1.11	36	0.2559	0.0244
diags.pc2_initiation	0.00	0.00	0.00	36	0.0004	0.0000
ukca	2.96	3.13	3.54	36	0.9102	0.0869
ukca.chemistry_alg	2.95	3.12	3.53	36	0.9060	0.0865
diags.ukca	0.01	0.01	0.01	36	0.0028	0.0003
diags.dynamics	0.01	0.01	0.01	36	0.0032	0.0003
diags.pmsl	0.00	0.00	0.00	36	0.0003	0.0000
diags.pressure_lev	0.00	0.01	0.01	36	0.0016	0.0002
gungho_driver.timestep	262.33	262.41	262.47	35	76.3127	7.4974
gungho_driver.finalise	4.53	4.72	7.55	1	1.3737	4.7237

mo-marqh · 2026-01-26T09:11:17Z

testing is complete, this is ready for Sci/Tech review @tommbendall

tommbendall

Thanks for doing this @mo-marqh

I spotted one unclosed timer and a typo (that wasn't yours). In the timer.txt that you posted, I noticed several 'control_*' calipers appear in gungho_transport_control. Could you also remove these?

Other than that should be good to go.

tommbendall · 2026-01-26T15:33:11Z

interfaces/physics_schemes_interface/source/algorithm/skeb_main_alg_mod.x90

      if (du_tot_skeb_flag) call du_tot_skeb%write_field('stochastic__du_tot_skeb')
      if (dv_tot_skeb_flag) call dv_tot_skeb%write_field('stochastic__dv_tot_skeb')
    end if
+    if ( LPROF ) call start_timing( id_diags, 'diags.skeb' )


Suggested change

if ( LPROF ) call start_timing( id_diags, 'diags.skeb' )

if ( LPROF ) call stop_timing( id_diags, 'diags.skeb' )

addressed in b70d7a6

tommbendall · 2026-01-26T15:38:43Z

interfaces/physics_schemes_interface/source/algorithm/skeb_ndisp_alg_mod.x90

+  if ( LPROF ) call start_timing( id_skeb, 'skeb.ndisp' )

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !!  1)  Initialize feilds and operators  !!


Suggested change

!! 1) Initialize feilds and operators !!

!! 1) Initialize fields and operators !!

Just because I spotted this typo while scrolling through!

addressed in b70d7a6

tommbendall · 2026-01-26T15:49:03Z

science/gungho/source/algorithm/transport/control/gungho_transport_control_alg_mod.X90

Suggested change

Can we remove all of the 'control_*' timers from this file?

addressed in b70d7a6

mo-marqh · 2026-01-27T08:55:28Z

thanks @tommbendall
i think i've addressed all of your suggestions

please confirm whether this is now okay by you?

developer tests ran and passed

Test Suite Results - lfric_apps - calipersPerformance25/run7

Suite Information

Item	Value
Suite Name	calipersPerformance25/run7
Suite User	mark.hedley
Workflow Start	2026-01-26T16:25:24
Groups Run	developer

Dependency	Reference	Main Like
casim	MetOffice/[email protected]	True
jules	MetOffice/[email protected]	True
lfric_apps	mo-marqh/lfric_apps@calipersPerformance25	False
lfric_core	MetOffice/lfric_core@aa32824	True
moci	MetOffice/[email protected]	True
SimSys_Scripts	MetOffice/[email protected]	True
socrates	MetOffice/[email protected]	True
socrates-spectral	MetOffice/[email protected]	True
ukca	MetOffice/[email protected]	True

Task Information

✅ succeeded tasks - 1106

tommbendall

Thanks again for doing this. I'm very happy to have these organised nicely.

This passes science review, so over to @oakleybrunt

oakleybrunt · 2026-01-27T10:58:17Z

I am currently doing this review and noticed that there is quite a lot of inconsistent use of the tik value - e.g. id_diags not being used for all callipers named diags.<name>.

Since the tik value is just the hash value for the name of the callipered region, it doesn't actually matter what this is called as long as it isn't overwritten. However, for consistency and readability of the timing code, I think that these should be standardised.

Taking the id_diags value as an example, using this signals to the person reading the code that the section they are viewing is used for diagnostics (usually quite obvious but it illustrates my reasoning). Similarly, any code that comes under id_alg can be easily searched for and identified as algorithm code with a simple grep.

This is just my view and I am open to discussing whether this is a convention we would like to follow or not. So please reply to this with your views on this.

mo-marqh · 2026-01-27T12:23:23Z

I am currently doing this review and noticed that there is quite a lot of inconsistent use of the tik value - e.g. id_diags not being used for all callipers named diags.<name>.

the naming of this integer value is just within the scope of each code section that is calling one or more timers.
I have tried to follow the convention that appeared to be followed in #80, where one single timer_index in scope is always id and multiple timer_indices are given id_ to differentiate them within the local scope.

I'm not keen to rename further timer_index integers in code committed in #80 within the scope of this PR, which is already large.
I'm not sure how to impose a convention which is consistent and clear.

I feel like the timer names are what should be searchable, and that using the tik integers as search items will always be fragile.

I do understand your desire for consistency @oakleybrunt , i just fear that it'll be difficult to deliver in this PR, and then hard to maintain.

oakleybrunt · 2026-01-27T12:29:30Z

I do understand your desire for consistency @oakleybrunt , i just fear that it'll be difficult to deliver in this PR, and then hard to maintain.

Thanks Mark, I see your perspective and perhaps trying to define a convention as part of this PR is too ambitious. I think that I will start a wider discussion on naming conventions for timer hash ids instead.

mo-marqh · 2026-01-27T12:38:30Z

I do understand your desire for consistency @oakleybrunt , i just fear that it'll be difficult to deliver in this PR, and then hard to maintain.

Thanks Mark, I see your perspective and perhaps trying to define a convention as part of this PR is too ambitious. I think that I will start a wider discussion on naming conventions for timer hash ids instead.

i think that consistency in the string identity naming for each timer is more useful & important.
that's a significant facet of this PR

oakleybrunt

As discussed, naming conventions for hash argument not covered in this PR. Otherwise, changes are good. Thanks - approved.

oakleybrunt · 2026-01-27T14:01:18Z

Would you resolve the merge conflicts please?

resolved

github-actions bot assigned mo-marqh Jan 22, 2026

mo-marqh added 11 commits January 22, 2026 17:02

implementation patch adoption

4372ee1

jules caliper naming consistency

bda0dc4

caliper naming consistency algorithm partial

6056507

caliper naming consistency algorithm residual

30ae589

caliper naming consistency phys diagnostics

1415d1a

caliper naming consistency interface socrates

82b646f

caliper naming consistency adjoint

8087104

caliper naming consistency gungho algorithm partial

a7edaf4

caliper naming consistency gungho remnant

a7327e1

caliper naming consistency linear

058c238

tidy up removals for test failures

5c9f1d0

mo-marqh mentioned this pull request Jan 22, 2026

Time calipers pi25 #79

Closed

27 tasks

mo-marqh force-pushed the calipersPerformance25 branch from c202c23 to 5c9f1d0 Compare January 22, 2026 18:18

mo-marqh requested review from oakleybrunt and tommbendall January 26, 2026 09:10

tommbendall requested changes Jan 26, 2026

View reviewed changes

review actions

b70d7a6

mo-marqh requested a review from tommbendall January 26, 2026 17:29

tommbendall approved these changes Jan 27, 2026

View reviewed changes

oakleybrunt approved these changes Jan 27, 2026

View reviewed changes

Merge branch 'main' into calipersPerformance25

896f917

Merge branch 'main' into calipersPerformance25

e841ed9

tommbendall mentioned this pull request Jan 27, 2026

Solver improvements #177

Open

29 tasks

Merge branch 'main' into calipersPerformance25

8ddd9ad

oakleybrunt requested review from matthewrmshin and ss421 as code owners February 2, 2026 10:08

oakleybrunt approved these changes Feb 2, 2026

View reviewed changes

oakleybrunt merged commit 5c6385c into MetOffice:main Feb 2, 2026
7 checks passed

jennyhickson added this to the Spring 2026 milestone Feb 3, 2026

mo-marqh mentioned this pull request Feb 4, 2026

Performance comparison and performance tests #214

Open

28 tasks

	if ( LPROF ) call start_timing( id_diags, 'diags.skeb' )
	if ( LPROF ) call stop_timing( id_diags, 'diags.skeb' )

	!! 1) Initialize feilds and operators !!
	!! 1) Initialize fields and operators !!

Conversation

mo-marqh commented Jan 22, 2026 • edited by oakleybrunt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Code Quality Checklist

Testing

trac.log

Test Suite Results - lfric_apps - calipersPerformance25/run6

Suite Information

Task Information

Security Considerations

Performance Impact

AI Assistance and Attribution

Documentation

PSyclone Approval

Sci/Tech Review

Code Review

Uh oh!

mo-marqh commented Jan 26, 2026

Uh oh!

mo-marqh commented Jan 26, 2026

Uh oh!

tommbendall left a comment

Choose a reason for hiding this comment

Uh oh!

tommbendall Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mo-marqh Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

tommbendall Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mo-marqh Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

tommbendall Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mo-marqh Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mo-marqh commented Jan 27, 2026

Test Suite Results - lfric_apps - calipersPerformance25/run7

Suite Information

Task Information

Uh oh!

tommbendall left a comment

Choose a reason for hiding this comment

Uh oh!

oakleybrunt commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mo-marqh commented Jan 27, 2026

Uh oh!

oakleybrunt commented Jan 27, 2026

Uh oh!

mo-marqh commented Jan 27, 2026

Uh oh!

oakleybrunt left a comment

Choose a reason for hiding this comment

Uh oh!

oakleybrunt commented Jan 27, 2026 • edited by mo-marqh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mo-marqh commented Jan 22, 2026 •

edited by oakleybrunt

Loading

oakleybrunt commented Jan 27, 2026 •

edited

Loading

oakleybrunt commented Jan 27, 2026 •

edited by mo-marqh

Loading